Skip to content

Experimental standard library#14249

Merged
ekpyron merged 1 commit intodevelopfrom
experimental-stdlib
Jun 6, 2023
Merged

Experimental standard library#14249
ekpyron merged 1 commit intodevelopfrom
experimental-stdlib

Conversation

@nikola-matic
Copy link
Copy Markdown
Contributor

@nikola-matic nikola-matic commented May 19, 2023

note: this is essentially a copy of @axic's #10639; due to the pragma changes, it was easier to simply start over instead of rebasing the original PR.

@nikola-matic nikola-matic changed the title [WIPExperimental standard library [WIP] Experimental standard library May 19, 2023
@nikola-matic nikola-matic force-pushed the experimental-stdlib branch 5 times, most recently from d288373 to 772fafd Compare May 19, 2023 13:57
@nikola-matic nikola-matic force-pushed the experimental-stdlib branch from 772fafd to d195269 Compare May 22, 2023 09:10
@ekpyron
Copy link
Copy Markdown
Collaborator

ekpyron commented May 30, 2023

make sure to resolve clashes between user/local and standard library files

Well, item 1 and a test for non-quoted imports not working for files will do that.

changelog

It's experimental solidity, there's no need.

One more thing though: all the assembly blocks will need to be marked as assembly ("memory-safe") these days (in review we need to recheck if they actually are, but they should be).

@ekpyron
Copy link
Copy Markdown
Collaborator

ekpyron commented May 30, 2023

Why not finish this during a few weeks by the way? This should kind of be our main focus.

@ekpyron ekpyron mentioned this pull request May 30, 2023
4 tasks
@nikola-matic
Copy link
Copy Markdown
Contributor Author

Well, item 1 and a test for non-quoted imports not working for files will do that.

Yup, the item was really a leftover from before we decided on the new import syntax

It's experimental solidity, there's no need.

We had a discussion last week while you were off, and had the idea to introduce an experimental section in the changelog; this is of course not set in stone, but we figured that there should at least be some form of minimal communication to the users that this stuff is in fact available. In any case, we can discuss it next Wednesday.

Why not finish this during a few weeks by the way? This should kind of be our main focus.

Sure, although there's some via-ir stuff I have left over as well.

Comment on lines +128 to +136
set<string> const& GlobalContext::stdlibIdentifiers()
{
set<string> static names {
"ecrecover",
"ripemd160",
"sha256"
};
return names;
}
Copy link
Copy Markdown
Collaborator

@cameel cameel May 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to me that it would be simpler and more robust to have this integrated into constructMagicVariables() rather than as a list maintained speparately. We'll need to remember to keep the list in sync when stuff is added to or removed from stdlib and it's easy to forget or make a mistake.

I see that GlobalContext is constructed at the beginning of the analysis but already after parsing. At this point we already know if we're in experimental mode so we could have a constructor parameter that sets a flag, which constructMagicVariables() would use to not include some of the magic variables.

This would also simplify the analysis because for example we would not need any special-casing for stdlib in DeclarationContainer to prevent these magic variables from getting resolved.

Copy link
Copy Markdown
Collaborator

@ekpyron ekpyron May 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't read through the code myself so far - but why exactly do we need this list anyways? The point in making these things stdlib-defined is that they're no longer magic variables...

Without looking at it, is this due to the rest of analysis breaking if it has declarations for magic variables? We'll likely swap out those parts of analysis soon anyways, so :-)... Hm... although name and type resolution would of course be nice to keep around at first, it's currently a mess of an implementation, but it works and will probably keep working for quite a while...

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I could tell from review, it's to remove the names provided by stdlib from global scope when experimental mode is enabled. Otherwise you could still use the corresponding global builtins.

We only really need it if we want to make the global built-ins unavailable gradually, as we add them to stdlib. We could instead make them all unavailable right away. Not sure if that's acceptable though - they include stuff like abi or type(), which won't make it into stdlib for quite some time.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly; I think making them unavailable now is the way to go (which makes the current implementation wrong, as you can still use the builtins even if you don't import from the standard library; in fact, I think it's even worse - these are implicitly imported from the standard library even if you don't explicitly import them, which makes this even worse). I think in general, this should be reworked prior to merging, even though the current implementation does work, if only by luck :). The list in GlobalContext should go as well. When I say I want to make the global builtins unavailable now - I don't actually mean all of them; rather, after parsing is completed (and we do parse the standard library files as well), we already know the names of exposed standard library functions, which means we can exclude only those from the global builtins, (if that makes sense).

Comment on lines +7 to +17
function f(uint256 a) external returns (bytes32) {
return sha256(abi.encodePacked(a));
}

function g(uint256 a) external returns (bytes20) {
return ripemd160(abi.encodePacked(a));
}

function h(bytes32 h, uint8 v, bytes32 r, bytes32 s) public returns (address addr) {
return ecrecover(h, v, r, s);
}
Copy link
Copy Markdown
Collaborator

@cameel cameel Jun 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test does not seem to be adding anything meaningful to what the other three test for these functions already do. Maybe it would be better to remove it?

By the way, is this little coverage really adequate for stdlib? I know the functions are simple and all, but still, it's the standard library. I'm assuming that what we have here is all we'll have, because I can't imagine anyone will have the patience to come back later and add proper tests for these things. This never happens in practice.

I'd prefer if we at least had some mechanism to test that the new functions give the same outputs as the existing built-ins on a bigger range or inputs, especially including some corner cases.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test is here only to cover different import mechanisms, i.e. import "std/cryptography.sol" vs import { sha256 } from "std/cryptography.sol"`, I can remove it if it's superfluous.

By the way, is this little coverage really adequate for stdlib?

No, these were just some sanity cases I added to the previous PR, since that one had only one test originally if I remember correctly.

I'd prefer if we at least had some mechanism to test that the new functions give the same outputs as the existing built-ins

I agree, although I'm not really sure how to go about this, since if pragma experimental solidity is enabled, we shouldn't have access to the builtins (even though currently we do), which means we'd either have to have this as a unit test where we compile one source unit that uses builtins, and the other that uses stdlib, preserve the return values, and then compare them. soltest/isoltest doesn't allow this, as far as I know.

Copy link
Copy Markdown
Collaborator

@cameel cameel Jun 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test is here only to cover different import mechanisms,

Ah, ok then. I did not notice that. I guess it's fine then, but I'd recommend changing its name to make it more obvious.

I agree, although I'm not really sure how to go about this

Well, we might build something specifically for this later (e.g. allow using current builtins in isoltest expectations), but for now I'd be satisfied simply with more coverage. We already have some tests for these 3 builtins, we could replicate them for experimental too.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why a command-line test? It seems to me like this would work perfectly fine as a syntax test. Is this maybe a left-over from some initial version that needed CLI flags to enable stdlib or something?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, there was never a cmdline flag to enable the standard library, it was always done via a pragma. This was a test case from the original PR, and the failed import throws a compiler error from CompilerStack, so I just assumed this couldn't be handled by a syntax test; I'd argue this isn't really a syntax test either, since the syntax itself is correct? In any case, I can try creating a syntax test for this and see how it works out?

Copy link
Copy Markdown
Collaborator

@cameel cameel Jun 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our syntax tests cover parsing too so that should not be a problem.

If it throws from CompilerStack, then yeah, that's the reason. I expected these to be properly caught by isoltest but maybe they're not. I now see it does not even have a proper error code.

In that case I'd question if we really want this to be reported this way. It does not have a location, even though we could easily provide one, and will drive users nuts :) I'm even wondering if this will not be reported as an ICE in Standard JSON. Well, maybe this is fine for the experimental version. It's something that will have to be ironed out sooner or later, before we make it non-experimental, but that's still way off in the future.

// We are using the free memory pointer.
let input := mload(0x40)
mstore(input, hash)
mstore(add(input, 32), v)
Copy link
Copy Markdown
Collaborator

@cameel cameel Jun 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function does not clean v.

Internal functions do not automatically clean their arguments so if there's anything in the higher bits of the value someone passes into this function, it will show when you access it in inline assembly and will affect the calculations. The other arguments are fine because they take up whole slots and cannot have dirty bits. v can.

That is, unless the precompile itself ignores those higher bits. This may or may not be the case, not sure - but in any case we should definitely have a test case covering this. And if it does ignore them I'd at least expect a comment pointing this out because it's not obvious.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This, I had no idea about, and it's what I'm currently looking into by comparing it with the IR code generator, but I don't really see anything special here compared to the other precompiles.

Copy link
Copy Markdown
Collaborator

@cameel cameel Jun 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The version that's currently in the codegen uses ABI encoder on the arguments, which takes care of the problem. Yours does not do that. Which is fine, using ABI encoding just for this seems like an overkill, but you still do need manual cleanup in that case.

See the cleanup example from my operator blog post: https://blog.soliditylang.org/2023/02/22/user-defined-operators/#parameter-cleanup. You should be able to create a repro for this problem by using ecrecover() the same way I use yoloDiv() there (note that this does not depend on operators and UDVTs in any way, you can replace them with free function calls and uint8).

Copy link
Copy Markdown
Collaborator

@cameel cameel Jun 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me know if you need any help with fixing this or creating a test, I can explain everything.

@nikola-matic nikola-matic force-pushed the experimental-stdlib branch from d195269 to eee44a9 Compare June 6, 2023 12:11
@nikola-matic nikola-matic force-pushed the experimental-stdlib branch from eee44a9 to 6a60bc2 Compare June 6, 2023 14:38
Change import syntax and cover with tests
@nikola-matic nikola-matic force-pushed the experimental-stdlib branch from 6a60bc2 to 47969ad Compare June 6, 2023 15:17
@nikola-matic nikola-matic changed the title [WIP] Experimental standard library Experimental standard library Jun 6, 2023
@nikola-matic nikola-matic marked this pull request as ready for review June 6, 2023 15:17
@ekpyron ekpyron merged commit 09038ce into develop Jun 6, 2023
@ekpyron ekpyron deleted the experimental-stdlib branch June 6, 2023 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants